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ABSTRACT 



. The application of conventional psychometric procedures to 
instructional product development' Is outlined* Selected non- 
psychometric requirements of dfevelo^ng and delivering effective 
, instruction ar'e described and<.the consequent implications fox: ^ ' 
psychometric procedures set forth. Present instructional develop- . 
ment technology is circumscribed in terms, of state-of-the-art / 

s 

capability. . , . . • 
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IffiASUREMENT CONS^pERATKSJS IN INSTRUCTIONAL PRODUCT-'DEVELPPMENT* 

'llobert BaRer ^ • , • .* 

^ %, • 

' >' * • 

The psychometric revolutioir.fehiit has. been ^oldering over the past 
decade and finally ignited in the "criterion- referenced test movetileht" 
will predictably spread throughout education during the next decade, 
aiid will generate consequences that go well beyond the boundaries of 
psychometry^ (Schutz, 1972) . Even now i^ is obvious that concern with ' 
psychometric dogma reflected in such questions as "Is the criterion-, 
referenced test just a special instance of thfe norm-referenced test?*^' 
and '•How can thi reliability of criterion-referenced tests be assessed?" 
is misplaced* Focusing on such question^ is, about as productive as. the 
programmed instruction research pf t^e^l960*s related to overt-covert 
and^ large step-small step issues* - ♦ 

Recent* infiftructional, research and development has dembnfetrated that 

/ * ' ' ' * • ^ * 

formal measurement can Indeed fulfill important roles *in producing 

/ , . * 

/ ' 
instructional programs to me^t prespecifled objectives* However^ full 

^gploitati on of th is role re q uires control ov e3^-n0n-pgyehem etric ^ as w eH 

as psychometric^variables. The purely technical aspects of pSychometry 

provide great, capability for instructional product development. jCon- ' 

ventional psychometric procedures can readily be adapted to generate 

measures which provide adequatehftases for those instructional decisions 

that can currently be made and effected* But this is inadequate to 

advance the state-of-the-art for Improving inistructional effectiveness. 



*Originally prepared as part of the^syi^sium "Evaluation of 
Instructional Systems: Current Problems Facing Developers," American 
Psychological Association Annual Meeting, Honolulti, Hawaii, Septem- 
ber 3, 1972. 
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The interface between psychometry and instructional development must 






include greater attention to instructiCnal decision algorithms that are ^ 






defined as>f unctions of achievement measures anchored systematically to 






the manipulable conditions that produced the achievement. Th^/considera-- 


■ / 


1 


tion will ^encompass not only the specifications and development of 
instruction but also the installation and continuing operation of 
instruction. The effectiveness of specified instructional decision 




algorithms is dependent upon well-defined assessment procedures that are 




• 


easily reflected in defined behavioral clashes- of interest and anchored 






in manipulable instructional determinants. 


• 




The manipulable determinants of achievement 'in developing instructional 






V* 

programs *ar6 materials and procedures . To be use^l in a development* 






context" tests must be designed and constructed in a manner that defines 
the explicit rules linking patterns, of test performance to behavioral 


• 

i 
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^ * i 

referents' anchored in sequenced^ instructional materials ajid procedures. 
Further, to be useful in an operating Instructional context tests must 


■ ->■ , J 


yoe configured ia such a way that a particular decision algorithm may 






be applied with little inconvenience. . 




X \ 


The testing requirements |ollowing from these conditions are manifold. 






I dnd the scientific and technological bases for getting op with it range 






; from adequate to non-existent. However, absence of these bases caniiot 


: 




halt development efforts. We must identify the immediately available 






resources fqr developing effective instruction arid move as quickly as 






possible to completion of first generation shelf .items, recognizing that ^ 
the items thus produced Vepresent only a beginning of "more .to come" 


- 


V 


from programmatic educational R&D currently in progress. ' ^■ 
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' The remainder of the paper will view selected psychometric requirements 
and strategies as they interfac'e with selected^ non-psychometric requiremenjts 
of de\)'elopiilg and delivering effective instruction. , The view will be 
f^bm within defined SWRL R&D activities and state-of-the-art capability. 

MATTERS THAT ARE WELL WETHIN SWRL STATE-OF-T HE-ART 

^ ' ' ' '■ 1—: ' 

The instructional development technology described in this section 
is readily available in shelf-^tem or easily adaptable form. ' 

Writing Instructional Objectives 

The "how-to" information for stating well-formed instructional 
objectives has been available for some time. A convenient recent 
synthesis of this informktion is contained, jji, the SWR^ Staff Development 
Compendium (Baker & Schutz, 1971). By reading this information, ak /• 
interested high school-graduate-equivalent person caii^acquire all of the 
information required to mdet; this condition. The time-consuming and/ 

« \ * * 

thought-challenging task of what outcoines^o prepare remains to be done. 

But this is a matter of doing the job, ^'rather thaij of not knowing how'; . 

h ■ ' — — — . — , — ^ - . . . - _ - I — ^ . — ■ - ■ ■ ' * <i 



When the job of preparing weXl-formed instructional outcomes has 
been completed, one is at best at the beginning rather than at th^ end 
of instructioi^^jLleffectivenegs. Buktihe beginning is firm, leather than 



wishfu|. 



Criterion-Referenced T^i^t' Construction ^ * ' 

Not; only' does instructional development necessitate prespecified' 
instructional outcomes, it also requires a means of assessing the 
attainment of theses outcomes. This involves test construction activity. 



To 'be minimally usefiil the tests must be specifically referenced to a 
.prespecif ied structure of achievement. To be maximally useful the tests 
must be specifically referenced to defined instructional materials. A^' 
consequent requirement is to define criterion behavidr in the sp*ecif icatio 
of the limits vf a population of responses called for in the instruction 
which defines the criterion behavior, rather than in a list of responses 

which exemplify it. This is not a new. concept; it was encompassed by. 

> 

earlier discussions of content validity (Lennon, 1956) and Bruner's (1960) 
discussion of the structure pf the subject matter knowledge. Hgyever, 
specific procedural cues fox meeting the requirement were not available 
until Hively (1963) introduced the^, "item form." 

The item_fprm and related processes provide a neat system for 
blueprinting tests that meet all of the requirements of the psychometric 
concept of :content validity and at the same time contribute to the , 
definition of the behavioral structure of the subject matter domain 
treated. A collection of item forms sequentially ordered, .together 
with the replacement sets for the variable elements, could adequately 
define a universe of content across specified outcome areas^. When such 
prQcedures are more generally exploited the impractlcality of constructing 
criterion-referenced tests for cqmpldx behavioral-content domains cited 
by Ebel (1971) is overcome. ^ 

Instructional Specifications ... 

* An item form defines classes o^ behavior, but it ddes not indicate 
how the behavior Is to be established > However, as strings of Item 
forms are prepared, it is po^^^sible to arrange j them into tentative 



seqi;endes that constitute an bpgrational "cognitive map" of a subject 
matter useful in guiding both instructional and e^aluational efforts. 

The "Instructional specifications" approach ^Sullivan, Baker & 
,Schut2, 1971) provides a set of procedures fot maping out the instructional 
and assessment sequences consistent with the item form. The instructional . 
specification (IS) is a convenient gliide to the development of effective 
instruction for a given instructional objective. A well constructed IS 
per instructional objective provides answers to the following questions: 

1. What outcomes (objectives) will the successful 
learner attain as a result of the instruction? 

2. What information (cue) xdLll be given the learner 

0 

t& increase his ability to perform the desired behavior? 

3. . What procedures (mastery items) will be used to . ^ ^ 
* provide for practice and assessment of the 

, desired behavior? 

■ 4. What are the characteristics (limits) of the 
correct responses or response choices for the 



. desired behavior and what are> the Characteristics 
of p^Lapsable .but incorrect responses? 
5. What relevant skills (entry skills) must the 
learner possess p^ior to the instruction fpr 
the present objective? ' • J 

Instructional programs thdfc are developed properly from a set of 
written IS's incorporate the instructional and assessment techniques 
directly into the program materials and procedures, thereby increasing 
the probability of high learner achievement , of the instructional 
objectives. 



The IS is primarily usefixl in specifying instruction prior to the 
development of materials and procedur|es. However, the structure aijd 
architecture of extant instruction and curricula areVseldom explicitly 
.stated. Postdictive analytic conventions (Sjaitlj, 1971) have been 
developed for use* in analyzing the instructional^ architecture of 
portions of instructional materials* Set and matrix notational con- 
ventions permit description of .extant material in terms of the following 
seven components: ^ ^ 

Elements' ; the phenomena to be described, compared, 
related, or otheirwise studied (e.g., objects, 
systetms, event^, groups). 
Variables ; the characteristics of properties of _ 
elements that^ are used to describe, compare 
^ and relate them (e.^g«, color, weight, cost),. 
Values: the terms, phrases, nxnabers, or other 
symbols which. are available for assignment 
to elements for a given variable (e.g., red, , 
4 pounds, 50o)* ' • /' • 

Describers t those values, of variables which are ' ^ 

assigned to particular elements. ^ . 

Observation/Measurement Procedures ; st*andard proce- ^ 
dures or algorithm used to assign values of 
variables. to particular elements (e.g., using 
a thermometer to measure the temperature of a 

■ \ - ' 

riqui<() ; . 



Relational Rule ; ^imles or algorithms 'vhicii specify 

describers forgone variable given describers for - 

~. ' > 2 " - . * ' ^ 

another- variable (e.g,, A « irr , all-i:he rectangular ' 

" r - * # „ * ~ . 

' ; . . 

• blocks are green), r _ - ' - • 

Correspondence Rules/ sets of rules used to^elate 

one set of lelements to anothex set of elements 

% ^ * - 

(e.g., the letter £ is pronotpced /p/) . 

Text Referenced Instructional Management Sygtems 

Tests and texts have traditionally been treated as independent units 

with given tests amenable to various^^teaUc&^-aad. tha. outcomes o"f^ instructing^ 

with a^given text assessable by various .tests,' It is possible however to 

produce tests referenced to a given text series. With the test directly 

coupled to the text a metns is. provided for determinig the extent "^to 

which specific outcomes are being attained by individual students after 

specified instruction. It .is also possible to. prepare supplementary 

practice materials r-eferenced to each criterion measure^for use wh^re 

adeq^uate proficiency^ is yet to be a4^tainedHi — Ihls-lntegra4:edl^se<tuence. of 

> ^ ' ' • > *• ' 

"text-test-troubleshobting material??** constitutes a simple instructional 

management, system^ which SWRL f or convenience has termed a Learning 

Mastery System (LMS) . < ^ ^ ' 

A prime limitation in producing, such systems is that current texts: 

rarely have clear statements of inattuctional outcomes. This limitation 

has been met by inferring the measureable outcomes associated with a 

■ • _ ' . i , ^. " 

given text. Although simple In structure and use,^ an LMS significantly 
expands the information available to. the teacher for instructional 
decisions. Each LMS iJrovides; . . 



.^A me^ns for student placement at th^ beginning or^^.^ ^ 

the scho^c^ year* ' * ^ - . , . 

• Criterion-referenced measures on three ^to eight ^ o 

instructional outcomes ten to f if teelT times 
during the year * ^ 

. Additional practice materials f&t the outcomes 

which iiave continuity throughout the text * ^ ' , . 
, . Mid-year and end-of-year evaluation measures, • 

< 

■ ^Multiple Matrix gampling . ' ' ' . - ' 

-The-specif-ic equations used in multiple matrix sampling provided by 

Lord (1960) and J.ord and Novick^ (1968) have been procedurally adapted ^ 

for implementation (Shoemaker, 1973) and applied to large scale group 

achievement assessident. Results to date indicate that parameters' 

.estimated through ^jnultiple matrix sampling and parame^«r^^btained through ^ 

testing all examinees ^n all items may be interpreted similarly,^/ Parameters 

* » ^ , t ' 

estimated through^multiple matrix sampling may be contrasted with any 

predetermined standard- defining the minimal level of acceptable achievements 
S tate-gf-the-Art St atistical Analyses 

^ ^ , -^^ ^ ^ ' y . 

The Laboratory's reseayctf' tod development activities require on-line 

access to large data files and considerable flexibility in manipulating, 

. - * - * > 

analyzing, and retrieving information. In addition to standard statis- 

tical and matrix manipulation utility packages, a cajxability has been ^ 

developed for t^e continuous upgrading, of an extensive library or 

computer prograra-buiJLding modules. This permits quJ.ck modification .<5f 

computer program -ftlpct ions xd.th a( minimum of reprogramming for new 



procedures defined by staff. 



' /' MATTERS THAT ARE ON THE LEADING EDGE OF SWRL STATE-OF-YHE-ART' 
. JhQ^*t^as and' activities 'described in this section include iteips 

' : \ ' ' . \ ■■ . •-' --^ '\"^ ' ■- ■/ 

•that while not quite availsible as, "stielf-i terns," will influence the 
"new** generation of §WKL instructional products. ' * 

* Quality Aggurance Systems J ^ 

ThB release of a SWRL-developed instructional program requires 

demonstration that it has been used succ^sfully to obatin prespecified 

• - * ' ' ' M 

levels of pupil performance. To provide a-^-eplicable means of insuring 

* ^ * . ' *' - " • 

that the program continues to function at these levels j> a set of 
procedures r^ef erred/ to as Quality Assurance (Hanson, 1972) has been 
developed; These procedures providife en-rpute^ itif omj^tion on various 
indicators of performance and pacing useful to teachers^ principals, 
and district adminis|ratqrs. Teachers have -benefited from.Qualitir 
Assutance because it 'provides information helpful in planning and pacing 
ipatructionai activ^^t^e^ throughout the school year. Principals and 
district personnel find Quality. Assurance .helps keep theia informed of 
the status of an instructional program in each cl^ss throughout the 
3choQj year*. Pupils also benefit because it provides teachers vith 

the assistance needed to coi^plete all instructional units and to ^'^^ 

\ - i 

achieve high performance on the major outcomes* 

■Integrated Instructional Information Systems " . \ ' 

Text referenced Instructional management systems assign the teacher 
^otal accountability for the attainment of instructional outcomes. While 
the teacher often passes the^^^ponsibillty on to the students^ and 



pccaslonally other school personnel, the teacher at 4)resent is the sole 
manager of ins traction • *r * » - , 

The confounding of the.teaoher, instructional materials, and 
instructional decisions in assessing accountability fails .to.recognize 
that the teacher sharejs responsibility for the instructional progress of 
Students with, administrators at thecschool and district levels and with 
parents. It is. possible .to* provide useful information -to earch of these 
groups. ^However, the mechanisms, for doing this are sufficiently complex 
to require 'automation of analysis and reporting functions. This is the 
scope of the SWRL Instructional Maftagement System (IMS) . , ^ ^ 

^ The SWRL Instructional Management System, operates in conjunction 
.jwith a developed ins true tifonal system such' as the SWRL/Ginn Kindergarten 
Prograip.or with, an application of the Teict Referenced Management System. 
Utilizing a variety of communication modes 'for input and output, reports 
for each category of individuals are specially designed to aggregate and 
synthesize the inf ormation ^in a manner that is understandable and compre- 
hensive,, consistent with need-jbo-know^ requirements of teachers, principal 
curriculum supervisors, district administrators, parents, students^ and 

development personnel (McManus, 1973). , / * 

* "'''-''It 

Program Fair Evaluation . . ^ . 

In the SWRL context "program fair" simply indicates that all 
assessment procedures are sy^stematically referenced to the particular 
objectives of the program and the stimulus content used in instruction 
related to the objectives. Shoemalcer (1972). has reviewed the state-of- 
therart in this area. These techniques provide fair appoximations for . 



"program fair" comparison^ of Instructional programs, ttie adet^uacy of 

^ ^ ! * ' ' ' ' - 

the approximations "can .oniy be assessed after the techniques have been . 

further exex^9lsed empirically; . ' - • * 

> - • . 
The Architecture of Instructional Programs - ' - . 

Th^ item form and % the instructional specif fcations- (IS) are useful 

'tools in instructional product development, but they are neither necessary 

nor sufficient to initiate or, advance a given product development effort. 

. " . . • * - ^ 

Sets of IS*s make it possible to define "trees" at an intermediate level 

of complexity above the. micro-^level of behavioral objective "twigs" 

byt below the macro-level of an architeetural framework. The ^architectural 

* framework Qf.'aii instructional program converts the "jungle" of Instruction 
"fnto an. orderly "forest" confi-gurationi 

Emulating established procedures -in the architecture of physical 

^structures, the architecture of Instruction can be conducted In stages 

•of schematic specifications, through preliminary specifications, to working 

specifications. Instructional architecture subsumes*^the planning of 

"skills" and "content" conventionally considered in. t^st ^design. ^ Scate- 

ments of instructional architecture are as yet few and far between. 

Examples, of preliminary sp.ecif ications can be found in Quellmalz (19i73X. 

An example of working specifications can be found in a*SWRL (1972) 

document prepared by Baker, drawing upon various previous SWRL paper^- 

«. ■ * * »* 

Instructional Development Control and Monitoring System (IDCMS) " 

IDCMS represents an integrateii hardware configuration presently 

being installed within the Laboratory facility.' It represents k ^ 

powerful tool for increasing the sophistication of educational research 

and development ^ctivJLties. Computer applications to behavior research 



have-ig^ically betSn restricted to statistical 'analyses of data callecte^ 
off-line* This type of ^requlremexxt can be handled by standard statistical 
an,d inatrlx manipulation utility packages* AltliougBnsuch ^ capability is 
Important, Laboratory product design requlr^ents include sttidies of real 
time intex-actions between subject and equipment*. Exploitation of IDCHS 
capabilitite will permit'' .on- line eiq>erlmentation involving complex event 
seque|ices^. variai>le media utiHzation, and real time test eontingencies » ' 
Figure 1 includes a block diagram of the IDCMS configuration. 



M&TTEBg THAT ARg BEYOND SWRL SIATE-OF-THE-ART - 
In an R&D context an instructional pr.oduct that completes all stages 



' of the developmeat cycj^e is considered final; the **now" generatlpn of the 

* » • • ^ • ' ' » \ 

product unashamedly represents the best that can presently.be delivered* 

^Howeyer, longjbefbre the "Aow" product has gone to market the outcomes 

- ' ' ' ' ^ ' \ ' , 

x>f progammatlc R£cD activity provide th& scientific and technological bases 



for the "new" generation* " Listed below are some'lteloiis that were they 
now even^"leading edge," the description of "new" generation products 
would likely be dramatically ^different* Yet, untilM:hey\ are classjed as 

elf items the "new" generation of Instructional products 
cannot be, expejpted to reflect th^*, * - — ^, 

' / -_!*, .Instructional database stxuctures In fields 
^ ' other than tnathematlcs and readings 
2*- Systemic structu?5es in the social domain. 
3* Cost-feasible automated interactive instructional 

V 

0 

hardwate/softwat^e systems* *' 



4» .Algorithms for prespecified instructional decision 

contingencies, — 
5. Qiiality control isystem? for. aspects of performance ^ 

other than qualitative attaitimeut^ tiie^..j3nd cost. ^ > 
That the development of these items will Involve measurement 
considerations is .clear. These considerations move far from such 
classical topics as validity, reliability, item analysis, norming, and 
other traditional tools of psychometric theory and practice, It^is . 
well to have these tools in the instructional development kit, jjut more 
sophisticated tools' are clearly needed, ' ' 



/ 
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